Finite Sample Guarantees for Quantile Estimation: An Application to Detector Threshold Tuning

In threshold-based anomaly detection, we want to tune the threshold of a detector achieve an acceptable false alarm rate. However, tuning is often non-trivial task due unknown output distributions. A that provides rate equivalent specific quantile distribution. Therefore, use estimators based on order statistics estimate threshold. The estimation quantiles from sample data has more than century-long tradition and provide three different distribution-free finite guarantees for class estimators. first Dvoretzky–Kiefer–Wolfowitz (DKW) inequality, second utilizes Vysochanskij–Petunin third exact confidence intervals beta These are then compared used in problem. We both simulated as well obtained experimental setup with Temperature Control Lab validate provided.